Online Knowledge Distillation with Diverse Peers

Distillation is an effective knowledge-transfer technique that uses predicted distributions of a powerful teacher model as soft targets to train a less-parameterized student model. A pre-trained high capacity teacher, however, is not always available. Recently proposed online variants use the aggregated intermediate predictions of multiple student models as targets to train each student model. Although group-derived targets give a good recipe for teacher-free distillation, group members are homogenized quickly with simple aggregation functions, leading to early saturated solutions. In this work, we propose Online Knowledge Distillation with Diverse peers (OKDDip), which performs two-level distillation during training with multiple auxiliary peers and one group leader. In the first-level distillation, each auxiliary peer holds an individual set of aggregation weights generated with an attention-based mechanism to derive its own targets from predictions of other auxiliary peers. Learning from distinct target distributions helps to boost peer diversity for effectiveness of group-based distillation. The second-level distillation is performed to transfer the knowledge in the ensemble of auxiliary peers further to the group leader, i.e., the model used for inference. Experimental results show that the proposed framework consistently gives better performance than state-of-the-art approaches without sacrificing training or inference complexity, demonstrating the effectiveness of the proposed two-level distillation framework.Comment: Accepted to AAAI-202

Accelerating Diffusion Sampling with Classifier-based Feature Distillation

Although diffusion model has shown great potential for generating higher quality images than GANs, slow sampling speed hinders its wide application in practice. Progressive distillation is thus proposed for fast sampling by progressively aligning output images of $N$-step teacher sampler with $N/2$-step student sampler. In this paper, we argue that this distillation-based accelerating method can be further improved, especially for few-step samplers, with our proposed \textbf{C}lassifier-based \textbf{F}eature \textbf{D}istillation (CFD). Instead of aligning output images, we distill teacher's sharpened feature distribution into the student with a dataset-independent classifier, making the student focus on those important features to improve performance. We also introduce a dataset-oriented loss to further optimize the model. Experiments on CIFAR-10 show the superiority of our method in achieving high quality and fast sampling. Code will be released soon

Cross-Layer Distillation with Semantic Calibration

Recently proposed knowledge distillation approaches based on feature-map transfer validate that intermediate layers of a teacher model can serve as effective targets for training a student model to obtain better generalization ability. Existing studies mainly focus on particular representation forms for knowledge transfer between manually specified pairs of teacher-student intermediate layers. However, semantics of intermediate layers may vary in different networks and manual association of layers might lead to negative regularization caused by semantic mismatch between certain teacher-student layer pairs. To address this problem, we propose Semantic Calibration for Cross-layer Knowledge Distillation (SemCKD), which automatically assigns proper target layers of the teacher model for each student layer with an attention mechanism. With a learned attention distribution, each student layer distills knowledge contained in multiple layers rather than a single fixed intermediate layer from the teacher model for appropriate cross-layer supervision in training. Consistent improvements over state-of-the-art approaches are observed in extensive experiments with various network architectures for teacher and student models, demonstrating the effectiveness and flexibility of the proposed attention based soft layer association mechanism for cross-layer distillation.Comment: AAAI-202

Domain-Specific Bias Filtering for Single Labeled Domain Generalization

Conventional Domain Generalization (CDG) utilizes multiple labeled source datasets to train a generalizable model for unseen target domains. However, due to expensive annotation costs, the requirements of labeling all the source data are hard to be met in real-world applications. In this paper, we investigate a Single Labeled Domain Generalization (SLDG) task with only one source domain being labeled, which is more practical and challenging than the CDG task. A major obstacle in the SLDG task is the discriminability-generalization bias: the discriminative information in the labeled source dataset may contain domain-specific bias, constraining the generalization of the trained model. To tackle this challenging task, we propose a novel framework called Domain-Specific Bias Filtering (DSBF), which initializes a discriminative model with the labeled source data and then filters out its domain-specific bias with the unlabeled source data for generalization improvement. We divide the filtering process into (1) feature extractor debiasing via k-means clustering-based semantic feature re-extraction and (2) classifier rectification through attention-guided semantic feature projection. DSBF unifies the exploration of the labeled and the unlabeled source data to enhance the discriminability and generalization of the trained model, resulting in a highly generalizable model. We further provide theoretical analysis to verify the proposed domain-specific bias filtering process. Extensive experiments on multiple datasets show the superior performance of DSBF in tackling both the challenging SLDG task and the CDG task.Comment: Accepted by International Journal of Computer Vision (IJCV

High-Pressure Synthesis of Barium Superhydrides: Pseudocubic BaH12

Following the discovery of high-temperature superconductivity in the La-H system, where for the recently discovered fcc-LaH10 a record critical temperature Tc = 250 K was achieved [Drozdov et al., Nature, 569, 528 (2019) and Somayazulu et al., Phys. Rev. Lett. 122, 027001 (2019)], we studied the formation of new chemical compounds in the barium-hydrogen system at pressures up to 173 GPa. Using in situ generation of hydrogen from NH3BH3, we synthesized previously unknown superhydride BaH12 with a pseudocubic (fcc) Ba sublattice, which was observed in a wide range of pressures from 75 to 173 GPa in four independent experiments. DFT calculations indicate a close agreement between the theoretical and experimental equations of state. In addition to BaH12, we identified previously known P6/mmm BaH2 and possibly BaH10 and BaH6 as impurities in the samples. Ab initio calculations show that newly discovered semimetallic BaH12 contains H2, H3 molecular units and detached H12 chains. Barium dodecahydride is a unique molecular hydride with metallic conductivity which demonstrates a superconducting transition around 20 K at 140 GPa in agreement with calculations (19-32 K). The interpretation of the multiphase XRD data was possible thanks to the development of new Python scripts for postprocessing the results of evolutionary searches. These scripts help quickly identify the theoretical structures that explain the experimental data in the best way, among thousands of candidates.Comment: Due to file size restrictions the supporting information file was uploaded to the Researchgate websit

Integrated bioinformatics identifies key mediators in cytokine storm and tissue remodeling during Vibrio mimicus infection in yellow catfish (Pelteobagrus fulvidraco)

IntroductionThe pathogenesis of Vibrio mimicus infection in yellow catfish (Pelteobagrus fulvidraco) remains poorly understood, particularly regarding the impact of infection with the pathogen on primary target organs such as the skin and muscle.MethodsIn this study, we aim to analyze the pathological intricacies of the skin and muscle of yellow catfish after being infected with V. mimicus using a 1/10 LC50 seven-day post-infection model. Furthermore, we have utilized integrated bioinformatics to comprehensively elucidate the regulatory mechanisms and identify the key regulatory genes implicated in this phenomenon.ResultsOur histopathological examination revealed significant pathological changes in the skin and muscle, characterized by necrosis and inflammation. Moreover, tissue remodeling occurred, with perimysium degeneration and lesion invasion into the muscle along the endomysium, accompanied by a transformation of type I collagen into a mixture of type I and type III collagens in the perimysium and muscle bundles. Our eukaryotic transcriptomic and 4D label-free analyses demonstrated a predominantly immune pathway response in both the skin and muscle, with downregulation observed in several cell signaling pathways that focused on focal adhesion-dominated cell signaling pathways. The upregulated genes included interleukins (IL)-1 and -6, chemokines, and matrix metallopeptidases (mmp)-9 and -13, while several genes were significantly downregulated, including col1a and col1a1a. Further analysis revealed that these pathways were differentially regulated, with mmp-9 and mmp-13 acting as the potential core regulators of cytokine and tissue remodeling pathways. Upregulation of NF-κB1 and FOSL-1 induced by IL-17C and Nox 1/2-based NADPH oxidase may have held matrix metallopeptidase and cytokine-related genes. Also, we confirmed these relevant regulatory pathways by qPCR and ELISA in expanded samples.DiscussionOur findings unequivocally illustrate the occurrence of a cytokine storm and tissue remodeling, mediated by interleukins, chemokines, and MMPs, in the surface of yellow catfish infected with V. mimicus. Additionally, we unveil the potential bidirectional regulatory role of MMP-9 and MMP-13. These results provide novel perspectives on the intricate immune response to V. mimicus infection in yellow catfish and highlight potential targets for developing therapies